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Abstract 

Higher order thinking skills (HOTS) is one of important aspects in education. Students with high level of higher 
order thinking skills tend to be more successful. However, do this phenomenon also happen in the learning of 
Mathematics? To answer this question, this research aims to study the relationship between HOTS and students' 
academic performance in Mathematics instruction. The research is conducted by occupying correlation research 
method on 41 students of mathematics education in university of Papua who had already completed 120 credits. 
The instrument in a test format for HOTS measurement has two main indicators; the critical thinking skill and 
creative thinking skill. Students are selected as research subjects, and are asked to do 9 questions of HOTS test in 
60 minutes. The holistic rubric is used to assess the higher order thinking skills of students. The results of 
research show that these two variables have a high value of correlation (r = 0.814) and the regression equation is 
grade point average = 2,105 + 0,017 HOTS. Both of statistics show that there is a significant relationship 
between HOTS and students' academic achievement. 
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1. Introduction 

Human thinking skills can be classified into two major groups; low-order thinking skills (LOTS), and higher 
order thinking skills (HOTS). LOTS are the first three aspects of taxonomy bloom, which are remembering, 
understanding, and applying. HOTS are the last three aspects of taxonomy bloom namely analyzing, evaluating, 
and creating (Moore & Stanley, 2010). In other words, HOTS is the highest part in Bloom’s taxonomy of 
cognitive domain. 

HOTS are important aspects in teaching and learning. Thinking skills are fundamental in educational process. A 
person’s thought can affect the ability, speed and effectiveness of learning. Therefore, thinking skills is 
associated with learning process. Students who are trained to think demonstrate a positive impact on the 
development of their education. Students with HOTS are able to learn, improve their performance and reduce 
their weaknesses (Yee, Othman, Yunos, Tee, Hasan, and Mohammad, 2011). 

According to Kings, Goodson, and Rohani (2013), HOTS are the ability to think that not only requires the ability 
to remember, but also higher capabilities. HOTS are student’s abilities that are activated when students encounter 
unfamiliar problems, uncertainties, questions, or dilemmas. Moreover Pogrow (2005) states that HOTS are 
valued because they are believed to prepare students better for the challenges both in advanced academic life and 
adult’s work and responsibility in daily basis. Therefore, HOTS can be used to predict the success of a student. 
Students who have good level of HOTS are expected to succeed in their studies later. 

Various efforts have been made by some researchers to improve HOTS students in some countries. Foong (2000) 
in Singapore conducted research on open-ended problems for higher order thinking in mathematics. In Georgia, 
Murray (2011) examines the implementation of higher order thinking in the middle school mathematics 
classrooms. The other research is conducted by Ghasempour, Kashefi, Bakar, and Miri (2012) in Malaysia on 
higher order thinking via mathematical problem posing tasks among engineering students, while Tajudin (2015) 
studies mathematical knowledge and higher order thinking skills for teaching algebraic problem solving in 
Turkey. 
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Furthermore, there are several other studies that investigate the relationship between HOTS and academic 
performance of students. Yee et al. (2011), concludes that there is a very low positive relationship between the 
level of Marzano HOTS with gender, academic achievement and socio economic status. Consequently, students 
should be assisted to acquire HOTS; either through the conventional teaching and learning environment or a self- 
instructional, individualized manual. On the other hand, Ramos, Dolipas, and Vilamor (2013) examines the 
relationship between HOTS and academic performance in physics of college students, and concludes that HOTS 
level on analysis, comparison, and evaluation significantly influence the physics performance of male students, 
while the HOTS level on analysis, inference, and evaluation significantly influence the physics performance of 
female student. Yoshida (2015) concludes that task-specific coaching rubrics enhance learners’ knowledge and 
understanding of curriculum development for higher-order thinking, and promotes learners’ skills to develop a 
curriculum for higher-order thinking. 

The problem arises is, is there is a relationship between HOTS and academic performance of students in 
mathematics instruction? How to measure HOTS of students who learn mathematics? What are the indicators of 
HOTS in mathematics instruction? Ramos et al. (2013), states that HOTS include skill such as 
creative and critical thinking, analysis, problem solving, and visualization. These skills involve 
categorizing items, comparing and contrasting ideas and theories, and being able to write about and solve 
problem. In the classroom, abilities and skills that include the use of HOTS are complex thinking that goes 
beyond basic recall of fact, such as evaluation and invention, enabling students to retain information and to apply 
problem-solving solution to real-world problems. 

Barak, David, and Uri (2007), stated that HOTS is composed of three components: critical, systems, and creative 
thinking, while according to Wang and Wang (2011), there are three main components in HOTS, i.e. critical 
thinking skills, thinking design and systems thinking. Based on these two statements, then the creative thinking 
skills proposed by Barak et al. (2007) similar to design thinking skills proposed by Wang and Wang (2011). Thus 
there are three components in HOTS, namely: (1) critical thinking skills, (2) creative thinking skills, and (3) 
systems thinking skills. 

Furthermore, Tanujaya (2014) in his analysis states that there are at least two indicators in HOTS, i.e. critical 
thinking and creative thinking skills. Moreover Tanujaya (2016) also asserts that there are nine factors that 
comprise HOTS in mathematics instruction; the use of mathematical concepts, the use of mathematical 
principles, impact predicting, problem solving, decision-making, working in the limits of competence, trying 
new things, divergent thinking, and imaginative thinking. Thus, there are at least nine items as test instruments 
used to measure HOTS. 

Various test instruments have been developed by the experts to be used as a selection tool in order to get 
prospective students. One such instrument is an instrument to measure developed by Tanujaya (2016). Moreover, 
he said that the HOTS instruments can be used to measure HOTS of students in mathematics instruction, because 
the instrument has acceptable validity and reliability 

Therefore, some questions raised are: is there a relationship between HOTS and student performance in 
Mathematics? How strong is the relationship between HOTS and student performance? Is the relationship linear? 
To answer this question, it is necessary to investigate the relationship between HOTS and GPA in mathematics 
instruction. 

2. Method 

The study is conducted using correlation research method. Gall, Gall, and Borg (2007) claim that correlation 
research method is a specific type of non-experimental design used to describe the relationship between two or 
more variables. The study is a research to determine the relationship and the strength of relationship between two 
variables without manipulating the variables. Correlation research refers to studies in which the purpose is to 
discover relationships between variables through the use of correlational statistics. 

Subjects for this research are 41 students of mathematics education from University of Papua, Manokwari West 
Papua Province, Indonesia. In this research, the subjects are mathematics education students who have 
completed 120 of 144 credits. In other words, they are most likely in their final year in Mathematics Education 
Department. The subject of research is selected using purposive sampling method. Purposive sampling, 
according to Etikan, Musa, and Alkassim (2016) is the deliberate choice of a participant due to the qualities the 
participant possesses. It is a non-random technique that does not need underlying theories or a set number of 
participants. Simply put, the researcher decides what needs to be known and sets out to find people who can and 
are willing to provide the information by virtue of knowledge or experience. 
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There are two variables used in this study; HOTS and GPA. HOTS are used as predictor variables, while the 
GPA as the criterion variable. Predictor variables according to Gall et al (2007) are measured before the criterion 
variable measured. The predictor variables are used to predict the criterion variables. 

HOTS students are measured using test instruments, while the students’ GPA obtained from mathematic 
education department, university of Papua. The instrument in a test format used in this study for HOTS 
measurement has two main indicators; the critical thinking skill and creative thinking skill. The instrument has 9 
variables, consisted of 5 variables for critical thinking and 4 variables for creative thinking. Five variables used 
for measuring critical thinking skill are conceptual comprehension, principles comprehension, impact prediction, 
problem solving, and decision-making. Meanwhile, the other four variables for creative thinking skill are 
working in competence limit, coping with new challenges, having both divergence thinking pattern and lateral 
thinking pattern (imagination). 

Students are selected as research subjects; they are asked to do 9 questions of the HOTS test during 60 minutes. 
The holistic rubric is used to assess the HOTS of students. The assessment is done based on the three major 
components; problem comprehension, procedure of problem solving, and the correct answers. 

The data obtained from this study are students’ test scores ranging from 0 to 111. This score is then converted 
into a value ranging from 0 to 100. The conversion is made because the range of scores commonly used is 
ranging from 0 to 100. Data resulted from the conversion was then statistically analyzed using correlation and 
regression analysis. Correlation is a statistical technique that measures direction and strength of the linear 
relationship between the two variables, while regression analysis is a statistical process for estimating the 
relationships among variables. The use of correlation and regression analysis in this research was to determine 
the relationship between HOTS and academic achievement of students in mathematics. Academic achievement 
used in this study is the GPA of Mathematics education students. 

According to Moore, Notz, and Fligner (2013), the calculation of correlation coefficient (r) used the formula: 


_ i 

n-1 



while estimating equation regression is done by using the formula: 


with slope: 


y — a + bx 


b = 



( 1 ) 

( 2 ) 

( 3 ) 


and intercept: 


a = y — bx (4) 

Furthermore, Best and Kahn (1998) stated that there are a number of ways to interpret a correlation coefficient. 
The inteipretation is depending on the research’s purpose and the circumstances that may influence the 
correlation’s strength. One method that was frequently presented crude criterion was presented on Table 1. 


Table 1. Criteria for the relationship between the two variables using coefficient correlation 


No. 

Coefficient (r) 

Relationship 

1 . 

0.00 - 0.20 

Negligible 

2. 

0.20 - 0.40 

Low 

3. 

0.40 - 0.60 

Moderate 

4. 

0.60-0.80 

Substantial 

5. 

0.80-1.00 

High to very high 


In order to simplify the calculations, the data was analyzed using SPSS program package. SPSS is statistical 
package for social sciences. Therefore, the results of the data analysis presented in the form of SPSS modified 
output, and other results of the study were included in the following section. 
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3. Result and Discussion 

3.1 Descriptive Statistics of Variables 

Descriptive statistics is numerical and graphic procedures to summarize a collection of data. Descriptive 
statistics helps to simplify large amount of data into a simpler summary. The data resulting from research in the 
form of some student’s descriptive statistics of variables HOTS and GPA are presented in Table 2 below. 

Table 2. Descriptive statistics of students’ HOTS and GPA 


Variable 

N 

Minimum 

Maximum 

Mean 

Std. Deviation 

HOTS 

41 

10.00 

90.00 

39.4878 

23.08909 

GPA 

41 

2.04 

3.87 

2.7700 

0.47794 


Table 2 shows that the HOTS scores are distributed from the lowest level (10) to the highest level (90). The same 
thing is also shown by the GPA data of students distributed from the lowest GPA (2.04) to a higher (3.87). In 
addition, the standard deviation of GPA and HOTS in the table also indicates that the two variables have a good 
variability. The variability of data is crucial in the analysis of data using statistical procedures. Thus, the data can 
be used for correlation and regression analysis. This was in accordance with the opinion of Tanujaya (2013), 
which states that the variability of the data in statistics is an important factor in the analysis of the research data, 
both the estimation and testing of the population parameter. 

However, the descriptive analysis has not yet shown distribution of data. Is data of HOTS shown normal 
distribution? Further statistical testing showed that the assumption of normality of the data required in the 
regression analysis was also fulfilled (See Figure 1). 



Figure 1. Normal plot of regression standardized residual dependent variable: GPA 


The plot (Figure 1) is a check on normality. Gupta (2007) said that check on normality can be divided into two 
main themes - relying on statistical tests or visual inspection. Statistical tests have the advantage of making an 
objective judgment of normality but are disadvantaged by sometimes not being sensitive enough at low sample 
sizes or overly sensitive to large sample sizes. As such, some statisticians prefer to use their experience to make a 
subjective judgment about the data from plots. Graphical interpretation has the advantage of allowing good 
judgment to assess normality in situations when numerical tests might be over or under sensitive but graphical 
methods do lack objectivity. 

The plotted points in Figure 1 tend to follow the straight line. So the data now appears to be suitable for simple 
linear regression. According to Gupta (2007), a normal plot of regression standardized residual dependent 
variable can be used to conclude that the data appears to be normally distributed as it follows the diagonal line 
closely and does not appear from the line. 

3.2 Correlation between HOTS and GPA 

Correlation is the relationship between two or more paired variables or two or more sets of data. Correlation is a 
number that used to measures direction and strength of the linear relationship between two variables. The degree 
of relationship is measured and represented by the coefficient of correlation. Correlation between variables 
HOTS and GPA as result of SPPS is displayed on Table 3. 
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Table 3. Correlation between variables HOTS and GPA 



HOTS 

GPA 

HOTS Pearson correlation 

1 

0.814" 

Sig. (2-tailed) 


0.000 

N 

41 

41 


Correlation is significant at the 0.01 level (2-tailled). 


The correlations table displayed Pearson correlation coefficients, significance values, and the number of cases 
with non-missing values. According to Best and Kahn (1998), correlation coefficients is always a number 
between -1 and 1. Values of correlation coefficients near 0 indicate a very weak linear relationship. The strength 
of the linear relationship increases as r moves away from 0 toward either -1 or 1. Values of correlation 
coefficients are close to -1 or 1 indicates that the points in a scatter plot lie close to a straight line. Positive r 
indicates positive association between the variables, and negative r indicates negative association. The extreme 
values, where r = -1 and r = 1 occur only in the case of a perfect linear relationship, or when the points lie 
exactly along a straight line. 

Table 3 shown that there is a strong positive relationship (r = 0.814) between the two variables, HOTS and GPA. 
Furthermore, on the test p-value, it shows a very strong evidence (p<0.001) to suggest that there is a linear 
correlation between the two variables, HOTS and GPA. Based on the correlation coefficient and the p value, it 
can be stated that the higher the HOTS, the higher the GPA will be. The higher the students’ HOTS, the higher 
the GPA they will get. 

3.3 Regression between HOTS and GPA 

Regression analysis generates an equation to describe the statistical relationship between one or more 
explanatory variables and the response. Therefore, regression analysis is used to model the relationship between 
a response variable and one or more explanatory variables. In order to develop a regression equation, SPSS 
produces two types of coefficients, i.e. standardized coefficients and unstandardized coefficients as presented in 
the Table 4. 


Table 4. Coefficient of the estimated regression model of GPA and HOTS 


Model 

Unstandardized Coefficients 

Standardized Coefficients 

t 

Sig. 

Beta 

Std. Error 

Beta 

1 (Constant) 

2.105 

0.088 


23.976 

0.000 

HOTS 

0.017 

0.002 

0.814 

8.754 

0.000 


The standardized coefficients are appropriate in multiple regressions when the explanatory variables were 
measured on different units. These coefficients are obtained from regression after the explanatory variables are 
all standardized. The idea is that the coefficients of explanatory variables can be more easily compared with each 
other as they are then on the same scale. In simple linear regression, they are of little concern. Therefore, the 
estimation of regression equation is done by using unstandardized coefficients (Best and Kahn, 1998). 

Based on Table 4, the intercept is 2.105 and the slope is equal to 0.017. Thus, the regression equation that stated 
the relationship between GPA and HOTS are as follows: 

GPA = 2.105 + 0.017 HOTS (5) 

The model of regression analysis stated that for each increase of one unit in HOTS, the GPA value was expected 
to increase by 0.017 units. This statement implied that if two people have different HOTS value of 10, then they 
would have different GPA of 0.17. Based on the regression equation, it can also be predicted that a student who 
has a GPA of 70, his GPA is therefore equal to 3.295. 

Furthermore, the standard errors on Table 4 are the estimation of the variability of the (unstandardized) 
coefficients and are used for significance tests for the coefficients. The t values and corresponding significance 
values are tests assessing the worth of the (unstandardized) coefficients. Based on Table 4 we can conclude that 
both test are highly significant (p<0.001), indicating that we have very strong evidence of need both the 
coefficients in regression model. Feasibility of the regression model also appears at the results of analysis of 
variance regression model as presented in Table 5. 
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Table 5. Analysis variance of Regression between HOTS and GPA 


Model 

Sum of Square 

Degree of freedom 

Mean Square F 

Sig. 

1 Regression 

6.055 

i 

6.055 76.633 

0.000 

Residual 

3.082 

39 

0.079 


Total 

9.137 

40 




Table 5 shows model regression with a value of F is equal 76 633, and p value <0.001. The value of less than 
0.05 implyes that there is sufficient evidence to reject the hypothesis (H 0 ). The null hypothesis tested in this 
study is two coefficients (slope and intercept) equal to zero. In other words, the coefficients do not equal to zero 
and can be used in the regression model. 

4. Conclusion 

Based on the results of research and discussion, we can conclude that there is a linear, positive and strong 
relationship between HOTS and the GPA of students. Students with high level of HOTS are expected to succeed 
in their next study in study program of mathematics education. Students who have high HOTS tend to get high 
GPA in mathematics instruction, whereas those with low HOTS tend to have low GPA. Therefore, the value of 
HOTS can be used as an indicator in the selection of new students. 

In order to thrive in learning mathematics, mathematics education students should have high level of HOTS. One 
of the most important things that can be done to improve students' HOTS is by revising textbooks used in 
mathematics learning in primary and secondary schools. The mathematics textbooks used in Indonesia should 
promote students’ critical and creative thinking. Examples and practice tests provided should be able to train 
students to think critically and creatively by using open-ended test. The open-ended test is a test used as an 
instrument in this study to measure students' HOTS. 
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Appendix A 

The Instrument to Measure Student’s HOTS 

1. Given two similar rectangles. The rectangle A is 4 cm wide and 8 cm long, while the rectangle B has an 
area of 288 cm square. Determine the length and width of the rectangle B. 

2. Let x + y = 100, where x and y are integers. Determine the maximum value of xy. 

3. Find two integers that are equal to 60. Five times the smallest number is equal to 12 plus the largest 
number. 

4. Mrs. Lusia is a craftswoman who daily produces two kinds of souvenirs. Capital souvenir A is Rp.300/unit, 
with a profit of 20%, where capital souvenir B is Rp.400/unit, with a profit of 30%. Every day she is only 
able to produce 60 pieces of souvenirs, and the number of souvenir B produced is 2 times the number of 
souvenirs A. What is the number of each souvenir produced by Mrs. Lusia? 

5. Mary wants to make lampshades from cylindrical paper (note the picture) with height 20 
cm and diameter 9 cm. Vanesa stated that the required length of paper is 28 cm. According 
to Valentina 29 cm, while according to Valdo 30 cm. Whose opinion is right? Why? 


6. Aurelia reported that there were 20 wheels on the parking lot. If in the parking lot there are only minibuses 
and motorcycles, what is the maximum number of minibuses in the parking lot? 

7. Currently the population is a problem that received the attention of the Indonesian government. In 1971 the 
population of Indonesia is 119,208,229 people, and in the year 2010 reached 237,641,326 people. How do 
you use math to address the problem of population growth in Indonesia? 

8. Given a set: B = {Roni, Sweeti, Pluto, Browni, Helmi, Seven}. What is the name of B? 

9. Use the following question to construct a mathematical problem corresponding to the question. Paul is the 
oldest brother of Benny. The ratio of the age of both is 9:5. In the next five years Paul ages four times half 
of Benny’s age. How old are they today? 
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Appendix B 

Rubric to Assess Student’s Answer 


No 

Assessment Criteria 

Score 

1 

Question Understading 


a 

Be able to determine all the factors being asked 

4 

b 

A maximum of one factor in question cannot be determined 

3 

c 

More than one factor in question cannot be determined 

2 

d 

Not able to determine all the factors asked 

1 

e 

Not trying to determine the factor being asked 

0 

2 

Answer Procedure 


a 

The answer procedure is done efficiently, well, and correctly 

4 

b 

The answer procedure is correct but not efficient 

3 

c 

There is one error in the answer procedure 

2 

d 

The answer procedure does not lead to an answer of the question 

1 

e 

Did not try to answer the question 

0 

3 

Correctness of Answer 


a 

The answer is correct 

4 

b 

There is at most one error in the answer 

3 

c 

More than one error on the answer 

2 

d 

Answer wrong 

1 

e 

No answer 

0 
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